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November  1,  2006 

MEMORANDUM  FOR  COMMANDER,  OPERATIONS  AND  TRAINING  BRANCH, 

HEADQUARTERS  UNITED  STATES  AIR  FORCE 
COMMANDER,  AIR  COMBAT  COMMAND 
DIRECTOR,  SIMULATOR  SYSTEMS  GROUP, 
AERONAUTICAL  SYSTEMS  CENTER 
DIRECTOR,  DEFENSE  CONTRACT  MANAGEMENT 
AGENCY 


SUBJECT:  Report  on  the  Acceptance  and  Surveillance  of  F-16  Mission  Training  Center 
Simulation  Services  (Report  No.  D-2007-008) 


We  are  providing  this  report  for  information  and  use.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

Comments  on  the  draft  of  this  report  conformed  to  the  requirements  of  DoD 
Directive  7650.3  and  left  no  unresolved  issues.  Therefore,  no  additional  comments  are 
required. 

We  appreciate  the  courtesies  extended  to  the  staff.  Questions  should  be  directed 
to  James  L.  Komides  at  (614)  751-1400  x21 1  or  Amy  J.  Frontz  at  (303)  676-7392.  See 
Appendix  E  for  the  report  distribution.  The  team  members  are  listed  inside  the  back 
cover. 


By  direction  of  the  Deputy  Inspector  General  for  Auditing: 


Paul  L'Granetto,  CPA 
Assistant  Inspector  General  and  Director 
Defense  Financial  Auditing  Service 


Department  of  Defense  Office  of  Inspector  General 

Report  No.  D-2007-008  November  1,  2006 

(Project  No.  D2005-D000FJ-0 171.001) 

Acceptance  and  Surveillance  of  F-16  Mission 
Training  Center  Simulation  Services 

Executive  Summary 

Who  Should  Read  This  Report  and  Why?  DoD  personnel  responsible  for  managing, 
executing,  and  overseeing  contracts  for  services,  including  aircraft  simulations,  should 
read  this  report.  It  discusses  the  need  for  better  acceptance  testing  and  contract 
surveillance  for  an  Air  Force  F-16  aircraft  simulation  training  services  contract. 

Background.  In  June  1999,  the  Air  Force  awarded  a  contract  to  Lockheed  Martin 
Integrated  Systems  to  acquire  F-16  aircraft  simulation  services.  The  Air  Force  planned  to 
use  the  simulation  services,  which  include  cockpit  simulators,  to  provide  mission  training 
to  its  F-16  pilots.  The  Contractor  was  required  to  develop,  deliver,  and  maintain  Mission 
Training  Centers  that  provided  simulation  services  in  accordance  with  Government 
approved  perfonnance  specifications  that  included  simulating  advanced  F-16  missions. 

Results.  The  Air  Force  accepted  substandard  Contractor  performance  and  paid 
$145  million  for  simulation  services  that  did  not  meet  critical  contract  specifications. 

The  Mission  Training  Centers  did  not  simulate  14  of  the  16  missions  specified  in  the 
contract  and  there  were  deficiencies  in  the  services  the  Air  Force  accepted.  Additionally, 
the  limited  service  was  impaired  by  repeated  computer  system  failures.  As  a  result,  the 
Air  Force  did  not  achieve  the  enhanced  training  capability  it  anticipated  and  pilots  were 
continuing  to  use  the  F-16  aircraft  to  perform  required  mission  training  at  a  cost  of  more 
than  $5,400  an  hour.  The  Air  Force  needs  to  analyze  the  existing  capabilities  of  each 
F-16  Mission  Training  Center  site  against  contracted  requirements  and  reduce  service 
payments  through  the  end  of  the  contract  in  June  2007  for  any  requirements  not  provided. 

The  Air  Force  is  in  the  process  of  developing  a  new  contract  for  future  F-16  simulation 
services  and  it  plans  to  spend  $582  million  to  obtain  services  under  the  new  contract  as 
early  as  2007.  The  Air  Force  needs  to  be  certain  that  problems  experienced  in  the  1999 
contract  for  F-16  simulation  services  are  avoided  in  the  future  by  perfonning  better 
acceptance  testing  and  contract  surveillance.  The  Director,  Simulator  Systems  Group, 
Aeronautical  Systems  Center,  needs  to  coordinate  with  Air  Combat  Command  and 
Defense  Contract  Management  Agency  personnel  to  require  contractors  to  demonstrate 
that  simulation  services  meet  all  statement  of  work  requirements  prior  to  service 
acceptance,  formalize  the  test  discrepancy  reporting  and  resolution  process  for  simulation 
service  contracts,  and  require  quality  assurance  surveillance  plans  to  be  developed  in 
conjunction  with  the  statement  of  work  for  all  simulation  service  contracts  as  required  by 
the  Federal  Acquisition  Regulation.  The  Commander,  Operations  and  Training  Branch, 
Headquarters  United  States  Air  Force  needs  to  update  Air  Force  instructions  to  identify 
roles  and  responsibilities  for  the  management  and  administration  of  simulation  service 
contracts  and  mandate  training  for  quality  assurance  personnel  perfonning  oversight  of 
simulation  service  contracts. 


Air  Force  Comments.  The  Assistant  Secretary  of  the  Air  Force  (Acquisition)  took 
exception  to  details  within  the  findings  of  the  report.  We  did  not  make  changes  to  the 
draft  report  as  a  result  of  the  Assistant  Secretary’s  comments.  However,  we  provided  a 
detailed  audit  response  to  the  Assistant  Secretary’s  comments  in  Appendix  D  of  this 
report.  The  Assistant  Secretary  concurred  with  the  recommendations.  The  Assistant 
Secretary  stated  the  Air  Force  will  work  with  the  Defense  Contract  Management  Agency 
to  monitor  the  contractor’s  performance  and  reduce  service  payments  if  services  do  not 
meet  contractual  requirements.  The  Assistant  Secretary  stated  no  additional  services  are 
planned  for  the  current  F-16  Mission  Training  Center  simulation  service  contract  and  the 
Air  Force  will  ensure  all  contractual  requirements  are  met  prior  to  accepting  future 
simulation  services.  The  Assistant  Secretary  stated  the  Air  Force  will  formalize  the 
discrepancy  reporting  and  resolution  process  in  future  simulation  service  contracts.  The 
process  will  include  the  date  each  discrepancy  is  identified,  tested,  and  resolved.  The 
Assistant  Secretary  stated  that  the  Defense  Contract  Management  Agency  is  currently 
drafting  a  quality  assurance  surveillance  plan  for  Mountain  Home  Air  Force  Base  and  the 
Air  Force  is  working  closely  with  the  Defense  Contract  Management  Agency  to  develop 
quality  assurance  surveillance  plans  for  future  simulation  services.  The  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  comments  were  fully  responsive  with  respect  to 
the  recommendations.  Therefore,  no  further  comments  are  required.  See  the  Finding 
Section  of  the  report  for  a  discussion  of  management  comments  and  the  Management 
Comments  section  of  the  report  for  a  complete  text  of  the  Secretary’s  comments. 

The  Director,  Current  Operations  and  Training,  Headquarters  United  States  Air  Force, 
concurred  with  the  finding  and  recommendations.  The  Director  is  revising  Air  Force 
Instruction  36-2251  “Management  of  Air  Force  Training  Systems”  to  more  clearly  define 
roles  and  responsibilities  of  personnel  who  manage  and  administer  simulation  service 
contracts.  The  Director  stated  that  Headquarters  Air  Combat  Command  is  doing  the 
same  for  Air  Force  Instruction  36-2248  “Operation  and  Management  of  Aircrew  Training 
Devices.”  The  Director  stated  the  revisions  of  Air  Force  Instruction  36-225 1  and  Air 
Force  Instruction  36-2248  will  also  mandate  training  for  Air  Force  personnel  when  a 
Defense  Contract  Management  Agency  contracting  officer  delegates  quality  assurance 
duties.  The  Director’s  comments  were  fully  responsive.  Therefore,  no  further  comments 
are  required.  See  the  Finding  Section  of  the  report  for  a  discussion  of  management 
comments  and  the  Management  Comments  section  of  the  report  for  a  complete  text  of  the 
Director’s  comments. 

Defense  Contract  Management  Agency  Comments.  Although  not  required  to 
comment,  the  Deputy,  Operations  Group,  Defense  Contract  Management  Agency  Dayton 
provided  comments  on  the  finding.  The  Deputy  stated  that  his  office  has  identified  for 
improvement  certain  internal  processes  in  areas  of  site  reporting  procedures,  lines  of 
communication,  and  training.  The  Deputy  also  stated  that  his  office  is  rewriting  the 
memorandum  of  agreement  with  the  Air  Force  and  has  taken  action  to  improve  their  level 
of  support  for  all  simulator  programs.  For  a  full  text  of  Defense  Contract  Management 
Agency  comments,  see  the  Management  Comments  section  of  the  report. 
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Background 


This  is  the  second  audit  of  an  existing  Air  Force  contract  for  F-16  Mission 
Training  Center  (MTC)  simulation  services.  We  perfonned  the  first  audit1  at  the 
request  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  and  examined  events  directed  by  Ms.  Darleen  Druyun  for  compliance 
with  Federal  acquisition  policy.  This  audit  examined  contracting  issues  that  did 
not  involve  actions  taken  or  directed  by  Ms.  Druyun. 

The  Air  Force  uses  several  Aircrew  Training  Devices  to  provide  F-16  pilot 
training.  Aircrew  Training  Devices  complement  training  available  in  the  aircraft 
and  provide  training  that  is  prerequisite  to  in-flight  performance.  The  training 
offered  in  the  devices  cannot  be  fully  conducted  in  the  aircraft  because  of  safety, 
airspace,  equipment,  and  security  restrictions. 

Historically,  F-16  pilots  used  low-fidelity  simulators  which  are  capable  of 
providing  stand  alone  training  but  not  F-16  mission  rehearsal  training.  The  Unit 
Training  Device  is  the  most  common  training  device  but  only  provides  marginal 
aircraft  weapons  training  and  is  incapable  of  training  important  missions 
including  suppression  of  enemy  air  defenses  and  deployment  of  high  speed 
anti-radiation  missiles. 

The  F-16  MTC  program  was  established  to  provide  simulation  service  including 
cockpits  that  simulate  the  fonn,  fit,  and  function  of  the  F-16  aircraft  with 
360  degree  visual  systems  and  high  fidelity  flight  and  weapon  simulations.  Each 
F-16  MTC  can  include  up  to  four  simulators  (cockpits)  and  their  associated 
instructor  operator  stations,  threat  stations,  modular  control  equipment 
workstations,  and  a  brief/debrief  station. 

The  F-16  MTC  is  intended  to  be  part  of  a  larger  distributed  mission  training 
system  that  integrates  E-3,  F-15,  and  F-16  weapons  system  simulators  to  perform 
mission  training.  Additionally,  the  F-16  MTC  includes  the  capability  to  train  in 
theatre-specific  environments  and  combat  situations,  thus  overcoming  constraints 
that  now  limit  war  preparation. 

The  program  is  managed  by  the  procuring  contracting  office,  Aeronautical 
Systems  Center  Headquarters  Simulator  Systems  Group  (fonnerly  the  Training 
Systems  Product  Group)  located  at  Wright-Patterson  Air  Force  Base  (AFB), 
Ohio.  Air  Force  Major  Command  personnel  manage  the  overall  F-16  pilot 
training  requirements  and  system  training  plan.  Air  Force  project  officers  and 
quality  assurance  representatives  provide  daily  oversight  of  the  F-16  MTC 
operations  and  are  responsible  for  monitoring  Contractor  performance. 

In  June  1999,  the  Air  Force  awarded  a  contract  with  a  ceiling  price  of 
$359  million  to  Lockheed  Martin  Integrated  Systems  (the  Contractor)  for  F-16 
MTC  simulation  services  using  commercial  item  acquisition  procedures.  The 
Contractor  agreed  to  develop  and  deliver  F-16  MTCs  that  provide  simulation 


1  DoD  Inspector  General  Report  No.  D-2006-065,  “Procurement  Procedures  Used  for  the  F-16  Mission 
Training  Center  Simulator  Services,”  March  24,  2006. 
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services  in  accordance  with  Government-approved  performance  specifications, 
and  to  keep  the  F-16  MTC  consistent,  or  concurrent,  with  the  F-16  aircraft.  The 
contract  called  for  a  limited  single-cockpit  simulation  service  (level  B)  and 
required  an  upgrade  on  the  single  cockpits  to  full  multi-ship  simulation  services 
(level  C). 

The  Air  Force  pays  the  Contractor  for  F-16  MTC  simulation  services  based  on  an 
hourly  service  rate  and  pays  100  percent  of  the  billed  hours  each  month  if  the 
Contractor  maintains  at  least  95-percent  operating  availability.  The  contract 
includes  a  complex  formula  for  calculating  service  availability  and  establishes  a 
point  system  based  on  the  availability  of  the  various  elements  of  the  F-16  MTC 
(See  Appendix  C  for  details). 

At  the  time  of  the  audit,  the  Contractor  was  providing  full  (level  C)  simulation 
service  on  14  cockpits  located  at  Shaw  AFB,  South  Carolina;  Mountain 
Home  AFB,  Idaho;  Spangdahlem  Air  Base,  Gennany;  and  Misawa  Air  Base, 
Japan.  The  Air  Force  paid  the  Contractor  $  145  million  for  F-16  MTC  services 
through  April  2006  and  the  existing  contract  is  scheduled  to  expire  in  June  2007. 

The  Air  Force  is  in  the  process  of  re-competing  the  requirement  for  F-16  MTC 
simulation  services  at  an  estimated  value  of  $582  million  and  expects  to  award  a 
contract  in  FY  2006. 


Objective 


The  audit  objective  was  to  determine  whether  the  management  and  oversight  of 
the  F-16  MTC  simulation  service  contract  ensured  that  services  provided  met 
contract  requirements.  See  Appendix  A  for  a  discussion  of  the  scope  and 
methodology  and  Appendix  B  for  prior  audit  coverage. 
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Acceptance  and  Surveillance  of  F-16 
MTC  Simulation  Services 

The  Air  Force  paid  $145  million  for  substandard  contractor  perfonnance 
including  simulation  services  that  did  not  meet  critical  contract 
specifications.  The  Mission  Training  Centers  did  not  simulate  14  of  the 
16  missions  specified  in  the  contract  and  there  were  deficiencies  in  the 
services  the  Air  Force  accepted.  In  addition,  the  limited  training  service 
provided  was  further  impaired  by  repeated  computer  system  failures. 

These  conditions  existed  because  the  Air  Force  did  not  ensure  that: 

•  all  required  testing  was  successfully  completed  prior  to  service 
acceptance,  and  that  discrepancies  identified  during  testing 
were  promptly  resolved; 

•  roles  and  responsibilities  for  contract  surveillance  were 
properly  defined,  and  that  personnel  monitoring  contract 
performance  were  adequately  trained;  and 

•  a  standardized  quality  assurance  surveillance  plan  was  in  place 
to  appropriately  penalize  the  contractor  for  limitations  with 
service  availability. 

As  a  result,  the  Air  Force  did  not  achieve  the  enhanced  training  capability 
it  anticipated  and  pilots  were  continuing  to  use  the  F-16  aircraft  to  perform 
required  mission  training  at  a  cost  of  more  than  $5,400  an  hour. 


F-16  MTC  Contract  Requirements 


Air  Force  contracting  personnel  clearly  outlined  the  specifications  for  F-16  MTC 
simulation  services  in  the  statement  of  work,  performance  specifications  list,  and 
training  task  list  that  were  a  part  of  the  contract  awarded  to  Lockheed  Martin 
Integrated  Systems  in  1999.  The  Air  Force  required  that  the  MTCs  be  capable  of 
training  all  procedures  specified  in  the  training  task  list.  Specifically,  the  training 
task  list  identified  more  than  900  pilot  training  tasks,  which  comprised 
16  required  aircraft  missions. 

Prior  to  delivery  and  start  of  service,  the  contractor  was  required  to  demonstrate 
that  the  F-16  MTC  provided  full  mission  training  capability  and  functioned  in 
accordance  with  contract  requirements.  The  Air  Force  specified  that  tests  were  to 
be  conducted  at  the  Contractor’s  facility  and  essential  test  discrepancies 
(problems  that  prevent  the  simulator  from  performing  specific  functions  essential 
for  mission  training)  identified  during  the  process  were  to  be  documented  and 
resolved.  The  contract  required  the  F-16  MTC  device  to  remain  in  plant  until 
essential  test  discrepancies  were  resolved  by  a  team  consisting  of  a  test  director, 
subject  matter  experts,  and  Government  representatives. 
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Testing  of  F-16  MTC  Simulation  Services 


The  Air  Force  did  not  ensure  that  all  required  testing  was  successfully  completed 
prior  to  acceptance  of  F-16  MTC  services.  The  Contractor  delivered  and  started 
multi-ship  (level  C)  simulation  service  at  Shaw  AFB  in  January  2003  even  though 
the  F-16  MTC  had  not  successfully  completed  in-plant  testing.  In  addition, 
numerous  essential  discrepancies  existed  that  prevented  adequate  mission  training 
and  the  process  for  reporting  and  resolving  discrepancies  was  never  formalized. 

In-Plant  Testing  Results.  The  Air  Force  performed  an  assessment  of  the  F-16 
multi-ship  service  at  the  Contractor’s  Akron,  Ohio,  facility  from  August  19  to 
September  6,  2002.  The  assessment  identified  209  discrepancies,  of  which  161 
were  considered  essential  discrepancies  that  had  to  be  corrected  prior  to 
successful  completion  of  the  assessment.  The  Air  Force  team  stopped  the 
assessment  testing  prior  to  completion  of  all  of  the  tests  because  of  system 
stability  problems.  During  the  second  week  of  testing,  less  than  25  percent  of 
missions  were  considered  fully  successful.  Therefore,  not  all  missions  and 
mission  elements  were  able  to  be  assessed.  The  test  team  decided  to  wait  until  all 
essential  discrepancies  were  corrected  to  complete  the  assessment. 

On  October  7,  2002,  the  Contractor  held  a  ready-to-ship  assessment  meeting  with 
Air  Force  test  officials.  The  Contractor  indicated  there  were  415  outstanding 
discrepancies,  which  included  the  209  identified  during  the  Air  Force  assessment. 
The  Air  Force  test  director  stated  that  a  mutual  decision  was  reached  in  the 
meeting  to  hold  off  shipping  the  F-16  MTC  until  essential  discrepancies  were 
completed  and  a  follow-up  training  assessment  could  be  completed.  However, 
the  Contractor  shipped  the  F-16  MTC  to  Shaw  AFB  in  January  2003  without 
correcting  essential  discrepancies  and  successfully  completing  the  required 
assessment.  Because  the  training  assessment  was  not  completed  and  not  all 
missions  and  related  elements  were  tested,  the  Air  Force  had  no  assurance  that  all 
essential  discrepancies  were  identified. 

Post-Delivery  Testing  Results.  The  Air  Force  test  team  performed  additional 
testing  at  Shaw  AFB  in  January  2003.  The  team  identified  1 12  discrepancies  that 
had  been  previously  identified  by  Contractor  and  Air  Force  test  personnel  but 
were  never  corrected.  In  addition,  the  team  identified  49  new  discrepancies. 

The  Air  Force  test  team  again  returned  to  Shaw  AFB  in  July  2003  to  determine  if 
previously  identified  discrepancies  had  been  corrected  but  experienced  a  large 
number  of  computer  system  failures.  The  team  reported  that  proper  resolution  of 
previously  identified  discrepancies  could  not  be  adequately  tested  until  system 
stability  was  established.  Through  limited  testing,  the  team  noted  that  previously 
identified  discrepancies  had  not  been  corrected.  The  Air  Force  test  team 
identified  the  test  discrepancy  reporting  process  as  a  problem  area  and 
recommended  that  the  process  for  resolving  test  discrepancies  be  fonnalized.  At 
the  time  of  the  audit,  Air  Force  officials  still  had  not  fonnalized  the  discrepancy 
reporting  process  nor  resolved  the  process  for  closure  and  resolution  of  test 
discrepancies. 
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Contract  Surveillance  Roles,  Responsibilities,  and  Training 


Roles  and  responsibilities  for  surveillance  of  the  contract  for  F- 16  MTC 
simulation  services  were  not  properly  defined  and  personnel  monitoring  contract 
performance  were  not  adequately  trained. 

Contract  Management  and  Execution.  Air  Force  Instruction  36-2248, 
“Operation  and  Management  of  Aircrew  Training  Devices,”  May  1,  1998, 
specifies  Air  Combat  Command  as  the  combat  air  forces2  representative  for 
managing  and  providing  policy  and  guidance  for  training  systems  common  to  the 
combat  air  forces.  The  Air  Combat  Command  Training  Support  Squadron  is 
identified  as  the  office  of  primary  responsibility  for  fielded  training  devices  but 
the  Training  Support  Squadron  did  not  have  a  role  in  the  management  and 
execution  of  the  contract  for  F-16  MTC  simulation  services.  This  role  was 
assumed  by  the  Aeronautical  Systems  Center,  Simulator  Systems  Group. 

A  memorandum  of  agreement  for  the  planning  and  testing  of  the  F-16  MTC 
contract  was  signed  in  April  2001,  almost  2  years  after  contract  award.  The 
memorandum  established  the  responsibilities  for  Aeronautical  System  Center,  Air 
Combat  Command,  the  29th  Training  Support  Squadron,  and  the  346th  Training 
Squadron.  The  memorandum  specifically  identified  the  combat  air  forces 
responsibilities  and  relationships  as  those  identified  in  Air  Force 
Instruction  36-2248.  However,  the  cited  instruction  did  not  address  the 
acquisition  of  commercial  simulation  services,  such  as  those  provided  by  the  F-16 
MTC  contract. 

Contract  Administrative  Services.  The  Defense  Contract  Management  Agency 
(DCMA)  is  the  DoD  Component  that  works  directly  with  Defense  suppliers  to 
help  ensure  that  supplies  and  services  are  delivered  on  time,  at  projected  costs, 
and  meet  all  performance  requirements.  A  memorandum  of  agreement  between 
the  Air  Force  Simulator  Systems  Group  and  DCMA  was  signed  in  August  1998  to 
establish  the  DCMA  Dayton,  Ohio,  office  as  the  single  DCMA  office  responsible 
for  contract  administrative  services  for  all  Simulator  Systems  Group  contracts.  In 
contrast  to  the  agreement,  the  Simulator  Systems  Group  retained  the  authority  for 
reviewing  and  approving  the  Contractor’s  monthly  invoices  for  F-16  MTC 
simulation  services.  This  resulted  in  confusion  as  to  the  specific  roles  and 
responsibilities.  For  example,  one  F-16  MTC  project  officer  was  sending  reports 
on  poor  contractor  performance  to  DCMA  instead  of  Simulator  Systems  Group 
officials  responsible  for  monitoring  payments  at  the  time. 

Training  of  Surveillance  Personnel.  Air  Force  F-16  MTC  quality  surveillance 
personnel  did  not  receive  adequate  training  to  perform  the  required  quality 
assurance  functions.  During  site  visits  to  Shaw  AFB,  Mountain  Home  AFB,  and 
Spangdahlem  Air  Base,  Air  Force  F-16  MTC  quality  surveillance  personnel 
stated  that  they  had  never  received  formal  training  on  quality  assurance  for 
training  devices  or  related  simulation  services.  Air  Force  Instruction  36-2248 


2  The  Combat  Air  Forces  are  comprised  of  the  following  Air  Force  Major  Commands:  Air  Combat 
Command,  United  States  Air  Forces  Europe,  Pacific  Air  Forces,  Air  National  Guard,  Air  Force  Reserve 
Command,  and  Air  Education  and  Training  Command. 
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provides  specific  information  on  an  annual  command-directed  training  device 
quality  assurance  workshop  that  includes  training  sessions  on  career  development, 
quality  assurance  policy,  and  contract  administration.  The  instruction  specifies 
that  representation  should  include  all  project  officers  and  quality  assurance 
representatives  responsible  for  aircrew  training  devices.  Air  Force  F-16  MTC 
quality  surveillance  personnel  at  the  sites  we  visited  informed  us  that  they  had  not 
been  able  to  attend  the  quality  assurance  workshop  or  obtain  comparable  training 
because  their  units  did  not  provide  the  necessary  funding. 


Quality  Assurance  Surveillance 


A  standardized  quality  assurance  surveillance  plan  was  not  prepared  and 
implemented  to  appropriately  penalize  the  contractor  for  limitations  with  service 
availability.  Because  a  quality  assurance  surveillance  plan  was  not  prepared,  pilot 
feedback  forms  used  to  grade  simulation  services  were  not  standardized  or 
consistently  completed,  and  the  methodology  used  to  determine  the  availability  of 
the  service  was  not  consistently  applied  and  validated. 

In  order  to  ensure  the  Government  receives  the  goods  and  services  it  has 
purchased,  a  mechanism  must  be  in  place  to  validate  that  the  goods  and  services 
delivered  meet  the  requirements  of  those  purchased.  This  mechanism  is  known  as 
a  quality  assurance  surveillance  plan.  Federal  Acquisition  Regulation  part  46 
“Quality  Assurance”  subpart  46.4,  “Government  Contract  Quality  Assurance,” 
states  that  a  quality  assurance  surveillance  plan  should  be  prepared  in  conjunction 
with  the  preparation  of  the  contract  statement  of  work  and  should  specify  all  work 
requiring  surveillance  and  the  method  of  surveillance. 

Completion  of  Pilot  Feedback  Forms.  Air  Force  MTC  quality  surveillance 
personnel  used  pilot  feedback  forms  to  monitor  Contractor  perfonnance.  If  a  pilot 
feedback  form  identified  a  deficiency  with  the  simulation  service  or  identified  the 
training  session  as  non-effective,  the  Contractor  should  have  been  appropriately 
penalized.  The  pilot  feedback  fonns  were  originally  developed  by  the  Contractor 
and  have  been  modified  several  times  by  Air  Force  personnel.  Each  F-16  MTC 
location  was  using  a  different  version  of  the  pilot  feedback  fonn. 

Pilot  feedback  fonns  were  not  always  completed  and  returned  to  the  Air  Force 
MTC  quality  surveillance  personnel.  For  example,  at  Shaw  AFB  during  a 
6-month  period  beginning  in  May  2005,  there  were  more  than  40  instances  where 
pilots  did  not  complete  and  return  feedback  forms.  In  addition,  pilot  feedback 
forms  were  not  returned  to  Spangdahlem  Air  Base  MTC  quality  surveillance 
personnel  for  1 1  out  of  19  days  of  simulator  operation  during  November  2005. 

Air  Force  MTC  quality  surveillance  personnel  cannot  adequately  measure 
Contractor  performance  without  the  pilot  feedback  forms. 

Surveillance  of  Simulator  Service  Availability.  The  Air  Force  did  not 
consistently  apply  and  validate  the  methodology  used  to  detennine  service 
availability  at  Shaw  AFB,  Mountain  Home  AFB,  and  Spangdahlem  Air  Base. 
Because  this  was  a  commercial  services  contract,  we  expected  that  the  Air  Force 
would  simply  pay  for  actual  time  spent  in  the  MTC  based  on  an  established 
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hourly  rate.  However,  the  contract  for  F-16  simulation  services  included  a 
complex  fonnula  for  calculating  service  availability  and  established  a  point 
system  based  on  the  availability  of  the  various  elements  of  the  F-16  MTC.  To 
illustrate  its  complexity,  details  on  the  service  availability  fonnula  from  the 
contract  are  provided  in  Appendix  C.  As  discussed  below,  the  formula  was  not 
consistently  applied  and  validated. 

Surveillance  Activities  at  Shaw  AFB.  The  project  officer  for  the  F-16 
MTC  located  at  Shaw  AFB  developed  his  own  quality  assurance  surveillance  plan 
because  one  did  not  exist  when  he  arrived  in  June  2003,  which  was  a  year  after 
limited  simulation  service  (level  B)  began  in  May  2002.  Pilot  feedback  forms 
were  reviewed  and  service  availability  was  calculated  as  part  of  the  daily 
surveillance  activities.  In  addition,  quality  surveillance  personnel  accumulated  a 
wide  array  of  information  on  the  F-16  MTC  utilization  including  mission  late 
starts,  cockpit  utilization,  training  time  loss  and  non-effective  mission  rates.  The 
information  was  reported  to  Aeronautical  Systems  Center  and  Major  Command 
officials  responsible  for  the  management  of  the  F-16  MTC  simulation  service 
contract  but  was  not  used  to  hold  the  Contractor  accountable  for  perfonnance. 

In  an  attempt  to  hold  the  Contractor  to  the  performance  specifications  of 
the  contract,  Shaw  AFB  MTC  personnel  changed  their  methodology  for 
scheduling  simulation  service  and  calculating  service  availability.  In  doing  so, 
Shaw  AFB  personnel  were  able  to  demonstrate  several  months  where  the 
simulator  service  did  not  meet  the  95-percent  availability  threshold  required  by 
the  contract.  However,  Contractor  personnel  continued  to  submit  monthly 
availability  percentages  that  were  higher  than  those  calculated  by  Shaw  AFB 
personnel.  During  the  period  of  June  2003  through  October  2005  there  were 
seven  instances  where  the  Shaw  AFB  project  officer  calculated  availability  for 
simulation  service  below  95  percent.  However,  on  five  of  the  seven  occasions 
payments  were  based  on  the  higher  availability  rates  calculated  by  the  Contractor. 

For  example,  in  December  2003,  the  Contractor  calculated  a  91 -percent 
availability  rate  for  F-16  MTC  simulation  services  at  Shaw  AFB.  The  Shaw  AFB 
project  officer  calculated  a  79-percent  availability  rate  for  those  same  services.  The 
contract  identifies  the  project  officer  as  the  final  authority  for  detennining 
availability,  yet  the  Contractor  was  paid  based  on  the  91 -percent  availability  it 
calculated.  Furthermore,  the  availability  formula  in  the  contract  allowed  for  the 
Contractor  to  be  paid  at  an  even  higher  rate.  When  service  availability  falls 
below  95  percent,  the  Air  Force  pays  the  invoice  based  on  a  calculated  percentage 
equal  to  the  actual  availability  percentage  divided  by  95  percent.  For  this 
example,  the  Air  Force  paid  96  percent  of  the  amount  the  Contractor  invoiced  for 
service.  Therefore,  even  though  the  Air  Force  calculated  availability  at 
79  percent  of  the  required  service,  the  Contractor  was  paid  for  96  percent  of  the 
amount  billed. 

Surveillance  Activities  at  Mountain  Home  AFB.  Surveillance  activities 
at  Mountain  Home  AFB  differed  greatly  from  those  performed  at  other  MTC  sites 
and  did  not  provide  effective  Contractor  oversight.  The  project  officer  at  the  F-16 
MTC  located  at  Mountain  Home  AFB  was  not  using  a  quality  assurance 
surveillance  plan  when  we  visited  the  site  in  November  2005. 


7 


At  Mountain  Home  AFB  minimal  quality  assurance  surveillance  was 
performed.  The  project  officer  in  place  during  our  November  2005  site  visit  was 
the  sixth  project  officer  since  initial  F-16  simulation  service  began  in 
October  2002.  The  project  officer  reviewed  the  Contractor’s  availability 
percentage  report  and  compared  it  to  the  pilot  feedback  fonns  to  determine 
mission  reporting  accuracy.  However,  there  were  instances  where  pilots 
identified  missions  as  non-effective  for  training  but  the  Contractor  reported  them 
as  effective.  For  example,  simulator  missions  flown  on  four  separate  occasions  in 
November  2004  were  identified  as  non-effective  training  by  the  pilots  for  various 
reasons.  In  contrast,  the  Contractor  reported  the  same  missions  as  effective. 
Furthermore,  from  June  through  August  2005,  no  surveillance  activities  were 
performed  because  the  project  officer  was  deployed.  A  new  quality  assurance 
representative  was  appointed  during  the  project  officer’s  deployment  but  was  not 
aware  of  what  surveillance  activities  were  required. 

Surveillance  Activities  at  Spangdahlem  Air  Base.  Surveillance 
activities  at  Spangdahlem  Air  Base  differed  from  those  performed  at  other  MTC 
sites  and  did  not  provide  effective  Contractor  oversight.  The  project  officer  at  the 
F-16  MTC  located  at  Spangdahlem  Air  Base  had  a  copy  of  the  quality  assurance 
surveillance  plan  developed  at  Shaw  AFB,  but  was  not  using  it  during  our  site 
visit  in  February  2006. 

At  Spangdahlem  Air  Base  the  quality  assurance  surveillance  perfonned  by 
the  quality  assurance  representative  consisted  of  a  tally  of  effective  and  non- 
effective  missions  flown  in  the  simulator  reported  to  the  wing  commander.  The 
quality  assurance  representative  did  not  use  this  information  to  determine  the 
accuracy  of  the  Contractor-reported  availability.  For  example,  pilots  reported  a 
two-ship  simulator  mission  flown  on  October  18,  2005,  as  non-effective  training. 
The  quality  assurance  representative  also  reported  the  mission  as  non-effective  in 
his  summary.  In  contrast,  the  Contractor  reported  all  missions  on  October  18th, 
2005,  as  effective. 

Although  the  quality  assurance  representative  reported  effective  and  non- 
effective  mission  data  to  the  wing  commander,  the  representative  did  not 
consistently  report  this  information  to  F-16  MTC  contract  management  officials. 
While  this  information  may  be  useful  for  the  wing,  officials  responsible  for  F-16 
MTC  contract  management  could  use  the  information  to  hold  the  contractor 
accountable  for  failing  to  meet  the  specifications  of  the  contract. 


Certification  of  F-16  MTC  Simulation  Capabilities 


The  contract  for  F-16  MTC  simulation  services  required  the  services  to  be 
integrated  into  the  F-16  ready  aircrew  program  and  the  respective  training 
syllabus.  However,  the  official  certification  perfonned  by  Air  Force  experts  did 
not  recommend  that  any  of  the  mission  events  specified  in  the  contract  statement 
of  work  be  certified  for  ready  aircrew  program  accreditation. 

Air  Force  Instruction  1 1-2F-16,  Volume  1,  “F-16  Aircrew  Training,” 

May  1,  1998,  establishes  the  minimum  Air  Force  F-16  pilot  training  qualification 
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standards.  The  F-16  pilot  training  program  is  designed  to  progress  pilots  from 
initial  qualification  training  to  mission  qualification  training  and,  finally,  to 
continuation  training.  Continuation  training  involves  the  basic  flying  skills  to 
ensure  safe  operation  of  the  aircraft  (non-ready  aircrew  program  training)  and 
specific  mission-related  training  required  to  accomplish  the  unit’s  assigned 
mission  (ready  aircrew  program  training). 

Air  Force  Instruction  36-2248  provides  guidance  for  certification  of  aircrew  and 
mission  simulators.  The  Air  Combat  Command  Training  Support  Squadron 
establishes  simulator  certification  policy  and  guidance  and  the  29th  Training 
Support  Squadron  implements  Air  Combat  Command  policy  and  manages  the 
certification  program.  New  training  devices  require  initial  certification  after  the 
device  is  installed  and  the  Air  Force  has  completed  acceptance  testing.  The 
certification  process  is  designed  to  ensure  that  training  devices  are  maintained  to 
their  design  configuration  and  provide  accurate  and  credible  aircrew  training  in 
accordance  with  Air  Force  instructions. 

The  first  official  simulator  certification  report  on  the  F-16  MTC  was  issued  in 
January  2005,  approximately  2  years  after  delivery  of  full  (level  C)  simulation 
service.  The  report  summarized  the  results  of  the  initial  certification  that  was 
conducted  in  October  2004  at  Shaw  AFB.  The  certification  report  stated  that  the 
F-16  MTC  provided  satisfactory  support  for  normal  and  emergency  procedures, 
instrument  tasks,  element  coordination  and  air-to-ground  procedural  training. 

The  report  stated  that  the  F-16  MTC  was  not  being  maintained  to  design 
configuration  and  performance  of  the  F-16  aircraft,  which  significantly  restricted 
training  effectiveness  for  mission  qualification  training  and  continuation  training. 
Additionally,  upgrade  and  continuation  training  were  degraded  by  errors  in  the 
visual  environment,  and  mission  rehearsal  training  was  not  possible  because  of 
the  lack  of  Distributed  Mission  Training  capability.  Observed  mission  reliability 
of  72  percent  and  simulator  certification  questionnaire  comments  revealed  a 
continuing  concern  about  mission  training  lost  because  of  system  failures  and 
reboots.  The  report  did  not  recommend  any  of  the  mission  events  specified  in  the 
contract  for  F-16  MTC  simulation  services  to  be  certified  for  ready  aircrew 
program  accreditation. 


Effects  on  Mission  Training 


The  F-16  MTCs  were  not  capable  of  providing  mission  training  in  accordance 
with  the  Air  Force  ready  aircrew  program  requirements.  Pilots  were  only 
obtaining  limited  training  for  non-ready  aircrew  program  requirements,  which 
could  be  obtained  in  existing  training  devices.  As  a  result,  the  Air  Force  has  not 
realized  the  enhanced  training  capability  it  anticipated  and  Air  Force  pilots  must 
continue  using  the  F-16  aircraft  to  perfonn  required  mission  training  at  a  cost  of 
more  than  $5,400  an  hour. 

Each  Air  Force  Major  Command  is  responsible  for  developing  training 
requirements  based  on  its  specific  mission.  Headquarters  Air  Combat  Command 
periodically  issues  tasking  messages  that  provide  guidance  on  which  ready  and 
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non-ready  aircrew  program  events  can  be  accomplished  in  aircrew  training 
devices.  In  its  December  2005  tasking  message,  Headquarters  Air  Combat 
Command  indicated  that  only  two  non-ready  aircrew  program  mission  events 
required  by  the  contract  (emergency  procedures  and  tactics)  could  be  trained  in 
the  F-16  MTC.  These  requirements  could  also  be  trained  in  older  aircrew  training 
devices  such  as  the  Unit  Training  Device.  None  of  the  14  specific  missions 
required  by  the  contract  (See  Table  1)  were  listed  as  trainable  in  the  F-16  MTC. 


Table  1.  Contracted  Missions  Not  Trainable  in  the  MTC 

1 .  Killer  Scout 

2.  Air  Interdiction 

3.  Close  Air  Support 

4.  Nuclear  Operations 

5.  Defense  Counter  Air 

6.  Offensive  Counter  Air 

7.  Basic  Fighter  Maneuvers 

8.  Air  Combat  Maneuvering 

9.  Combat  Search  and  Rescue 

10.  Medium  Altitude  Operations 

11.  Weather  Approach  Procedures 

12.  Night  Vision  Goggle  Operations 

13.  Suppression  of  Enemy  Air  Defenses 

14.  Low  Altitude  Navigation  and  Targeting  Infrared  for  Night 


Summary 


The  Air  Force  did  not  properly  and  effectively  manage  the  F-16  MTC  simulation 
service  contract  and  has  continued  to  pay  for  simulation  service  that  never  met  the 
specified  training  requirements.  Authorities  did  not  make  the  Contractor  correct 
essential  test  discrepancies  and  system  stability  problems  and  demonstrate  that  the 
MTC  could  perform  mission  training  prior  to  shipment.  In  addition,  roles  and 
responsibilities  for  contract  surveillance  were  not  properly  defined  and  personnel 
monitoring  contract  perfonnance  were  not  adequately  trained.  Furthermore,  the 
inconsistent  surveillance  of  daily  MTC  operations  did  not  appropriately  penalize 
the  Contractor  for  limitations  with  simulation  services.  The  Air  Force  penalized 
the  Contractor  by  reducing  the  service  payments  by  $561,000  at  three  MTC  sites, 
which  represented  less  that  one  percent  of  the  $145  million  total  payments.  We 
believe  the  penalties  have  been  insignificant  and  did  not  suit  the  serious 
non-conformance  issues  that  the  Air  Force  experienced. 
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Management  Comments  on  the  Finding  and  Audit  Response 


Summaries  of  management  comments  on  the  finding  and  our  audit  response  are  in 
Appendix  D. 


Recommendations  and  Management  Comments 


1.  We  recommend  that  the  Director,  Simulator  Systems  Group, 

Aeronautical  Systems  Center,  coordinate  with  Air  Combat  Command  and 
Defense  Contract  Management  Agency  personnel  to: 

a.  Analyze  the  existing  capabilities  of  each  F-16  Mission  Training 
Center  site  against  contracted  requirements  and  reduce  service  payments 
through  the  end  of  the  contract  in  June  2007  for  any  requirements  not 
provided. 

Air  Force  Comments.  The  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
concurred  with  the  finding  and  recommendation.  The  Air  Force  agreed  to  reduce 
service  payments  to  the  contractor  if  the  F-16  Mission  Training  Center  simulation 
services  do  not  meet  contractual  requirements.  The  Air  Force,  in  association  with 
the  Defense  Contract  Management  Agency,  will  monitor  the  contractor’s 
performance  to  ensure  the  Air  Force  is  receiving  all  contractually  required 
services. 

b.  Require  contractors  to  demonstrate  that  simulation  services  meet 
all  statement  of  work  requirements  prior  to  accepting  the  service. 

Air  Force  Comments.  The  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
concurred  with  the  finding  and  recommendation.  The  Air  Force  stated  that 
additional  services  are  not  planned  for  the  current  F-16  Mission  Training  Center 
simulation  service  contract.  However,  for  future  simulation  service  contracts,  the 
Air  Force  will  ensure  all  contractual  requirements  are  met  prior  to  accepting 
service. 


c.  Formalize  the  test  discrepancy  reporting  and  resolution  process  for 
simulation  service  contracts.  The  process  should  include  a  description  of 
each  discrepancy  (problem  that  prevents  the  simulator  from  performing 
specific  functions  essential  for  mission  training)  and  the  date  each 
discrepancy  is  identified,  tested,  and  resolved. 

Air  Force  Comments.  The  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
concurred  with  the  finding  and  recommendation.  The  Air  Force  stated  the  request 
for  proposal  for  the  follow-on  F-16  Mission  Training  Center  simulation  service 
contract  identifies  detailed  requirements,  including  those  recommended  above,  for 
a  formalized  deficiency  reporting  system. 
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d.  Develop  quality  assurance  surveillance  plans  in  conjunction  with 
the  statement  of  work  for  all  simulation  service  contracts  as  required  by 
Federal  Acquisition  Regulation. 

Air  Force  Comments.  The  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
concurred  with  the  finding  and  recommendation.  The  Air  Force  stated  that  the 
Defense  Contract  Management  Agency  is  currently  drafting  a  quality  assurance 
surveillance  plan  for  the  F-16  Mission  Training  Center  site  at  Mountain  Home  Air 
Force  Base.  The  Air  Force  is  working  closely  with  the  Defense  Contract 
Management  Agency  to  develop,  in  accordance  with  the  Federal  Acquisition 
Regulations,  quality  assurance  surveillance  plans  for  the  follow-on  F-16  Mission 
Training  Center  simulation  service  contract. 

2.  We  recommend  that  the  Commander,  Operations  and  Training  Branch, 
Headquarters  United  States  Air  Force  update  Air  Force  instructions  to: 

a.  Identify  roles  and  responsibilities  for  the  management  and 
administration  of  simulation  service  contracts. 

Air  Force  Comments.  The  Director,  Current  Operations  and  Training, 
Headquarters  United  States  Air  Force,  concurred  with  the  finding  and 
recommendation.  The  Director  is  revising  Air  Force  Instruction  36-225 1 
“Management  of  Air  Force  Training  Systems”  to  more  clearly  define  the  roles 
and  responsibilities  of  personnel  who  manage  and  administer  simulation  service 
contracts  throughout  all  major  commands.  Headquarters  Air  Combat  Command 
is  doing  the  same  for  the  combat  air  forces  in  Air  Force  Instruction  36-2248 
“Operation  and  Management  of  Aircrew  Training  Devices.”  The  planned 
completion  for  these  revisions  is  December  31,  2006. 

b.  Mandate  training  for  quality  assurance  personnel  performing 
oversight  of  simulation  service  contracts. 

Air  Force  Comments.  The  Director,  Current  Operations  and  Training, 
Headquarters  United  States  Air  Force,  concurred  with  the  finding  and 
recommendation.  The  revisions  to  Air  Force  Instruction  36-225 1  and  Air  Force 
Instruction  36-2248  will  mandate  training  for  Air  Force  personnel  when  a 
Defense  Contract  Management  Agency  contracting  officer  delegates  quality 
assurance  duties. 

Defense  Contract  Management  Agency  Comments.  Although  not  required  to 
comment,  the  Deputy,  Operations  Group,  Defense  Contract  Management  Agency 
Dayton  provided  the  following  comments  on  the  finding.  The  Deputy  stated  that 
Defense  Contract  Management  Agency  Dayton  has  identified  for  improvement 
certain  internal  processes  in  areas  of  site  reporting  procedures,  lines  of 
communication,  and  training.  The  Deputy  stated  that  his  office  is  currently 
rewriting  the  memorandum  of  agreement  with  the  Air  Force  and  has  taken  action 
to  improve  their  level  of  support  for  all  simulator  programs.  For  a  full  text  of  the 
Defense  Contract  Management  Agency  Dayton  comments,  see  the  Management 
Comments  section  of  the  report. 
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Appendix  A.  Scope  and  Methodology 


This  audit  was  performed  as  a  result  of  conditions  that  came  to  our  attention 
during  our  audit  of  “Procurement  Procedures  Used  for  F-16  Mission  Training 
Center  Simulator  Services”  (Report  No.  D-2006-065).  The  original  audit  was 
performed  at  the  request  of  the  acting  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics.  An  internal  study  was  commissioned  in 
November  2004  in  response  to  admissions  by  the  Principal  Deputy  Assistant 
Secretary  of  the  Air  Force  (Acquisition  and  Management)  from  1993  to  2002  that 
indicated  the  Principal  Deputy  may  have  allowed  personal  interests  to  affect  her 
judgment  on  acquisition  decisions.  During  our  prior  review  we  identified  a 
number  of  issues  with  the  Air  Force  management  and  oversight  of  the  contract  for 
F-16  MTC  services.  For  this  audit,  we  determined  whether  the  management  and 
oversight  ensured  that  services  provided  met  contract  specifications. 

We  reviewed  the  Federal  Acquisition  Regulation,  Air  Force  Instructions,  and  the 
F-16  MTC  simulator  service  request  for  proposal,  statement  of  work,  perfonnance 
specifications,  and  contract  clauses.  We  reviewed  F-16  MTC  simulation  service 
pilot  feedback  data  from  three  of  the  four  F-16  MTC  facilities. 

We  interviewed  Headquarters  Air  Combat  Command  personnel,  including  the 
Chief  and  Deputy  Chief  of  the  Flight  Operations  and  Training  Division,  Air  and 
Space  Operations  Directorate,  and  personnel  from  the  Distributed  Mission 
Operations  Branch,  Directorate  of  Requirements.  We  also  interviewed  personnel 
from  the  Air  Combat  Command  Training  Support  Squadron,  and  the  29th 
Training  Support  Squadron  of  the  Air  Force  Warfare  Center  53rd  Wing  53rd  Test 
Management  Group.  We  interviewed  Aeronautical  System  Center  personnel 
including  the  Chief  of  Contract  Management,  Contracting  Officer,  Program 
Manager,  and  General  Counsel.  We  interviewed  the  project  officers  and  quality 
assurance  representatives  at  three  of  the  four  F-16  MTC  locations.  We  also 
interviewed  personnel  from  DCMA. 

We  perfonned  this  audit  at  Headquarters  Air  Combat  Command,  the  Air  Combat 
Command  Training  Support  Squadron,  the  29th  Training  Support  Squadron  at 
Eglin  AFB,  Florida,  Headquarters  Aeronautical  System  Center,  DCMA  Dayton, 
the  20th  Fighter  Wing  at  Shaw  AFB,  the  52nd  Fighter  Wing  at  Spangdahlem  Air 
Base,  and  the  366th  Fighter  Wing  at  Mountain  Home  AFB.  We  performed  this 
audit  from  November  2005  through  August  2006  in  accordance  with  generally 
accepted  government  auditing  standards.  The  scope  of  the  audit  was  limited  in 
that  we  did  not  visit  or  interview  personnel  from  the  Misawa  Air  Base,  35th 
Fighter  Wing  F-16  MTC  location.  In  addition,  the  scope  of  the  audit  was  limited 
in  that  we  did  not  review  the  managers’  internal  control  program  because  it  was 
not  an  announced  objective  of  the  audit. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Government  Accountability  Office  High-Risk  Area.  The  Government 
Accountability  Office  has  identified  several  high-risk  areas  in  DoD.  This  report 
provides  coverage  of  the  “DoD  Contract  Management”  high-risk  area. 
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Appendix  B.  Prior  Coverage 


During  the  last  5  years,  the  Government  Accountability  Office  (GAO),  the 
Department  of  Defense  Inspector  General  (DoD  IG),  and  the  Air  Force  Audit 
Agency  have  issued  four  reports  discussing  the  F-16  MTC,  and  three  reports 
discussing  surveillance  of  service  contracts.  Unrestricted  GAO  reports  can  be 
accessed  over  the  Internet  at  http://www.gao.gov.  Unrestricted  DoD  IG  reports 
can  be  accessed  at  http://www.dodig.mil/audit/reports.  Air  Force  Audit  Agency 
reports  can  be  accessed  over  the  Internet  at  http://www.afaa.hq.af.mil. 


GAO 


GAO  Report  No.  GAO-05-274,  “Opportunities  to  Improve  Surveillance  on 
Department  of  Defense  Service  Contracts,”  March  17,  2005 


DoD  IG 

DoD  IG  Report  No.  D-2006-065,  “Procurement  Procedures  Used  for  F-16 
Mission  Training  Center  Simulator  Services,”  March  24,  2006 

DoD  IG  Report  No.  D-2006-010,  “Contract  Surveillance  for  Service  Contracts,” 
October  28,  2005 


Air  Force 

Air  Force  Audit  Agency  Report  No.  F2006-0018-FDE000  “Distributed  Mission 
Operations,  52d  Fighter  Wing,  Spangdahlem  AB,  Germany,”  25  January  2006 

Air  Force  Audit  Agency  Report  No.  F2005-0039-FBN000  “F-16  Simulators, 
366th  Fighter  Wing,  Mountain  Home  AFB  ID,”  2  May  2005 

Air  Force  Audit  Agency  Report  No.  F2004-0051-FDM000  “Flight  Simulator 
Utilization,  20th  Fighter  Wing,  Shaw  AFB  SC,”  5  May  2004 

Air  Force  Audit  Agency  Report  No.  F2002-0044-WN0000  “Service  Contracting 
Quality  Assurance  Evaluation  Program,  Air  Research  Laboratory,  Kirtland  AFB 
NM,”  27  August  2002 
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Appendix  C.  Simulator  Service  Availability 

Accounting 

Contract  Clause  A-016.  Service  Availability  Accounting 


A.  Level  A  Availability  Accounting 

1.  The  Level  A  system  is  segregated  into  the  following  elements: 

ACC  F-16  Level  A  Elements 
Cockpit  with  IOS 
Visual 

Combat  environment 

2.  The  cockpit  with  IOS  and  either  the  Visual  or  the  Combat  environment  must  be 
available  for  Availability  to  be  non-zero. 

a.  All  three  elements  being  available  are  scored  as  100%  availability. 

b.  The  cockpit  plus  either  of  the  other  two  elements  is  scored  as  66%  availability. 
An  element  is  non-available  from  the  time  that  the  Government  first  reports  the 
outage.  Failure  of  the  contractor’s  problem  reporting  system  does  not  constitute  a 
failure  to  notify.  An  element  becomes  available  again  upon  successful 
completion  of  repair. 

c.  Monthly  availability  is  calculated  according  to  the  following  fonnula: 

#  hours  at  100%  *  100  +  #  hours  at  66%  * 

Lvl  A  Availability  (%)  =  ( - ) 

Total  number  of  hours  in  month 

d.  The  total  number  of  hours  in  a  month  are  number  of  days  in  a  month  *  24  hours 
a  day.  For  example,  January  has  3 1  days,  so  it  has  31  *  24  =  744  hours. 

B.  Level  B  and  Level  C  Availability  Accounting 

1.  The  F-16  Block  40/50  system  is  segregated  into  the  following  system  elements: 


ACC  F-16  MTC 

Elements 

Points 

Brief/DB  System 

2  room 

2  ( 1  each) 

IOS 

4  stations 

4(1  each) 

Pilot  Stations* 

4  cockpits 

4  ( 1  each) 

Manned  Threats 

4  stations 

4(1  each) 

Mission  Control  Equip 

1  station 

1 

Local  Network 

1  network 

1 

Long  Haul  Connect 

1  system 

5 

Observation  Area 

1  room 

1 

Total  Elements  Available 

22 

15 


*Pilot  stations  include  the  synthetic  environment  to  support  the  scheduled 
mission. 

NOTE:  These  segments  may  be  adjusted  based  on  contractor  selected  design  of 
the  system.  For  example,  if  a  separate  threat  environment  exists  to  support  each 
cockpit,  a  threat  environment  element  may  be  added  and  counted  like  4  cockpits 
or  manned  threats. 

2.  The  MTC  shall  have  95%  availability.  Each  of  the  above  elements  will 
contribute  the  points  indicated  toward  the  monthly  service  availability 
calculation: 


Available  Required  Elements  -  Not  Mission  Capable  Elements 

Availability  (%)  =  ( - - — - - - - )  *  100 

Required  Elements 

a)  Required  Elements.  Required  elements  are  identified  on  the  final  daily 
schedule  (normally  1600  the  day  prior)  presented  to  the  contractor.  Schedule 
changes  coordinated  after  a  final  schedule  is  established  will  not  be  charged 
against  the  contractor  if  the  contractor  is  unable  to  support  late  request.  However, 
additional  points  can  be  achieved  for  a  training  mission  by  supporting  late 
schedule  changes  (see  paragraph  2.d  below  and  examples).  If  these  elements  on 
the  final  daily  schedule  are  classified  Not  Mission  Capable  Elements,  these 
elements  will  be  excluded  from  the  definition  of  required  elements. 

b)  Available  Required  Elements.  An  element  is  considered  available  if  the 
element  was  used  in  the  mission  and  provided  or  supported  effective  mission 
training  for  the  primary  trainees  or  objectives.  If  the  element  is  down,  in  a 
degraded  condition,  or  not  able  to  be  used  to  support  the  mission  and/or  objective 
due  to  failure  of  another  element,  the  element  is  considered  not  available.  (Ex:  If 
the  cockpit  stations  fail,  it  is  very  unlikely  that  any  elements  can  be  considered 
available.)  If  team  training  is  scheduled  and  the  team  is  non-effective,  all 
elements  scheduled  by  the  team  shall  be  considered  non-effective.  The  crew  or 
instructor  responsible  for  accomplishing  the  mission  will  determine  mission 
effectiveness.  If  long-haul  connections  are  disrupted  due  to  problems  at  remote 
sites  (e.g.,  power,  phone  lines)  there  will  be  no  decrement  to  the  mission  points. 

NOTE:  The  basis  for  the  effective/non-effective  decision  will  be  patterned  after 
wing  policy  for  accomplishing  effective  training  flights  in  the  aircraft.  Crews  will 
attempt  alternate  missions  in  an  attempt  to  get  effective  training;  however,  basic 
qualification  type  missions,  emergency  procedure  practice  and/or  instrument 
missions  will  not  normally  substitute  for  team  mission  training  unless  a  team 
member  specifically  has  that  mission  type  as  an  identified  training  requirement. 

c)  Not  Mission  Capable  Elements.  An  element  is  considered  “Not  Mission 
Capable”  (NMC)  when  it  is  down  for  a  period  exceeding  one  day.  This  NMC 
category  protects  the  government  from  the  circumstance  of  being  unable  to 
schedule  a  desired  mission  because  the  mission  requires  the  use  of  an  NMC 
element,  and  the  element  is  known  to  be  NMC.  For  example,  two  instructor 
stations  are  down  for  an  extended  period  of  time.  If  the  user  only  schedules  two 
instructors  due  to  this  condition,  the  contractor  would  continue  to  get  100% 
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availability  without  the  NMC  provision.  Thus,  any  mission  conducted  during  a 
period  that  any  element  is  NMC  will  loose  element  points  for  NMC  elements. 

d.)  Any  missions  requested  by  the  user  and  supported  by  the  contractor  after  the 
final  daily  schedule  is  presented  to  the  contractor  are  credited  to  the  required 
elements  available  (numerator)  of  the  availability  formula  and  zero  points  are 
added  to  the  required  elements  (denominator).  If  late  requested  elements  are  only 
partially  effective  or  are  not  effective,  the  contractor  is  not  penalized  in  any  way; 
however,  no  points  for  the  mission  will  be  included  in  the  availability  accounting 
period. 

3.  The  on-site  Project  Officer  (PO)  is  the  final  authority  for  detennining  whether 
the  device  meets  the  mission  requirements.  The  MTC  contractor  and  PO  shall 
meet  as  required,  but  not  less  than  weekly,  to  resolve  the  accounting  of 
availability  factors.  The  ANG  POs  will  meet  with  the  contractor  no  less  than 
monthly.  Availability  shall  be  calculated  on  a  monthly  basis  but  a  review  of  the 
factors  will  be  accomplished  weekly  while  the  events  are  still  current.  In  the 
event  of  a  disagreement  between  the  Government  and  the  contractor's  accounting 
of  mission  availability  data,  the  decision  of  the  PO  shall  stand  with  the 
contractor's  position  documented  at  the  point(s)  of  disagreement.  Complete 
details  of  events/problems  that  affect  the  training  schedule  including  facility 
conditions  and  any  other  factors  affecting  the  availability  of  the  device  shall  be 
documented  by  the  contractor,  certified  by  PO  and  attached  to  the  contractor’s 
service  billing  invoice. 

4.  Each  month  the  cumulative  elements  available  for  all  missions  (available 
required  elements  minus  NMC  elements)  will  be  divided  by  the  cumulative 
elements  required  for  all  missions  to  calculate  the  monthly  availability 
percentage.  This  monthly  availability  percentage  will  then  be  multiplied  by  the 
hours  available  for  that  month  for  billing  purposes.  If  no  missions  are  scheduled 
or  flown  in  any  given  month,  the  availability  rating  will  be  100%  as  long  as  the 
contractor  staffs  the  MTC. 

5.  Under  no  circumstances  shall  the  availability  percentage  be  more  than  100%  in 
any  month. 

6.  Availability  will  not  be  calculated  until  operational  service  starts. 

7.  Availability  for  a  day  that  no  elements  are  scheduled  or  used  will  be  100%; 
however,  zero  points  will  be  added  to  both  elements  of  the  formula,  0  scheduled, 

0  used. 

8.  Availability  for  level  A  and  B  service  will  be  a  subset  of  the  MTC  elements 
using  the  same  scheduled  versus  effective  formula. 

The  following  examples  are  provided  for  clarification: 

Example  One: 

Force  Protection  Mission  Commander  Upgrade  Training:  Long  haul  DMT  with 
4  x  F-15C  at  Eglin,  AWACS  at  Tinker,  4  x  F-16C  cockpits  the  F-16  site. 
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Elements  Required/Scheduled  Points 

1  Briefing  system  1  point 

2  IOS  2  points 

4  Pilot  Stations  4  points 

4  Threat  Stations  4  points 

1  Mission  Control  Station  1  point 

Local  network  1  point 

Mission  Observation  1  point 

Debrief  System  1  point 

1  Long  Haul  Connect  5  points 

1  debriefing  system  1  point 

Elements  required/scheduled  21  points 


Case  I:  All  required  elements  are  logged  effective  by  the  crew,  the  availability 
rating  is  21/21  for  the  scheduled  mission. 

Case  II:  One  pilot  station  in  the  second  element  lost  the  visual  display  system 
10  minutes  after  the  mission  started.  The  second  element  “aborted”  and  had  no 
reasonable  alternate  mission.  The  second  element  instructor  station  was  no  longer 
required.  The  primary  mission  was  conducted  with  a  2-ship  element  and  logged 
as  effective  for  the  mission  upgrade  pilot  because  the  syllabus  allowed  a  2-ship 
lead  for  this  specific  mission.  One  IOS  and  two  cockpits  became  non-effective, 
therefore,  3  elements  are  non-effective  for  a  score  of  1 8/2 1 . 

Case  III:  The  mission  commander  upgrade  pilot  station  failed  for  loss  of  visual 
display  system  15  minutes  after  the  mission  started.  The  DMT  wide  mission  was 
not  stopped  and  the  L-15  flight  lead  at  Eglin  took  over  the  force  protection  lead. 
F-16  cockpits  were  swapped  and  the  mission  restarted  15  minutes  later  and 
attempted  as  a  two  ship  but  timing,  other  minor  delays  and  mission  flow  did  not 
allow  the  F-16  upgrade  pilot  to  accomplish  the  required  training;  therefore,  Non 
Effective.  No  effective  mission  training  was  accomplished  in  the  period.  The 
availability  rating  is  0/2 1 . 

Example  Two: 

One  element  (2-ship)  is  scheduled  for  a  DMT  SEAD  mission  and  the  second 
element  is  flying  local  low  level  and  range  familiarization  ride  with  two  threat 
stations  for  support.  The  local  mission  requested  no  MTC  briefing 


Elements  Points 

1  Briefing  system  1  point 

2  IOS  2  points 

4  Pilot  Stations  4  points 

4  Threat  Stations  (2  for  each  mission)  4  points 

1  Local  network  1  point 

1  Long  haul  Network/Setup  5  points 

2  Debriefing  systems  2  points 

Elements  required/scheduled  20  points 


Case  I:  All  required  elements  are  logged  effective  by  the  crew  for  Missions  #1 
and  #2.  The  availability  rating  is  20/20  for  the  scheduled  missions. 
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Case  II:  The  DMT  mission  was  non-effective  due  to  a  failure  in  the  MTC 
Contractor's  equipment  inside  the  MPOP.  One  DMT  mission  pilot  and  the 
instructor  stayed  and  did  a  single  ship  emergency  procedure  mission  to  prepare 
for  an  upcoming  qualification  requirement.  The  local  mission  went  as  scheduled. 
Loss  of  one  cockpit,  two  threat  stations,  and  the  long  haul  network  (8  element 
points)  were  deducted  for  an  availability  of  12/20.  If  the  DMT  mission  failure  is 
determined  to  be  out  of  the  control  of  the  MTC  Contractor  (ex:  O&I  connectivity 
issues),  the  loss  would  not  be  counted. 

Case  III:  The  DMT  mission  goes  as  planned  but  the  second  element  is  a  no-show. 
Availability  is  20/20. 

Example  Three: 

No  mission  is  scheduled  for  the  period  on  the  final  daily  schedule.  The  user 
requests  to  add  a  4  ship  local  mission  requiring  the  following  after  the  final 
schedule: 


Elements  Required/Not  Scheduled  Points 

1  Briefing  System  1  point 

1  IOS  1  points 

4  Pilot  Stations  4  points 

Combat  Environment  1  point 

1  Debriefing  System  1  point 

Local  network  1  point 

Elements  Required/Not  scheduled  9  points 

Elements  Required/Scheduled  0 


Availability  is  9  points  for  0  points  required.  9  will  be  added  to  the  available 
elements  (numerator)  for  the  cumulative  period.  No  points  will  be  added  to  the 
required  (denominator)  elements.  For  example,  if  the  contractor  points  for  the 
cumulative  period  was  390/412  (94.6%),  the  new  score  would  be  399/412 
(96.8%). 

C.  Availability  Accounting  Applicable  to  All  Orders 

1 .  This  section  C  is  applicable  to  any  order  placed  under  this  contract:  Level  A, 
Level  B,  or  Level  C  training. 

2.  The  system  or  element  will  be  counted  as  available  if  downtime  is  time  lost  due 
to  excusable  delays  as  defined  in  FEDERAL  ACQUISITION  REGULATION 
52.212-4(f). 

3.  Availability  of  95%  or  better  will  be  billed  at  100%  of  the  hourly  rate.  If 
availability  were  to  drop  below  95%,  the  Government  would  pay  a  fraction  of  the 
hourly  rate.  That  fraction  would  be  actual  availability  divided  by  95%. 
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Appendix  D.  Air  Force  Management  Comments 

on  the  Finding  and  Audit  Response 


This  Appendix  contains  a  summary  of  management  comments  on  the  finding  and 
our  audit  response.  See  the  Management  Comments  section  of  the  report  for  a 
completed  text  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
comments. 

Air  Force  Management  Comments  on  Certification  of  F-16  MTC  Simulation 
Capabilities.  The  Assistant  Secretary  of  the  Air  Force  (Acquisition)  disagreed 
with  the  finding  on  the  certification  of  F-16  MTC  Simulation  Capabilities  and 
cited  page  8  of  the  draft  report  that  states,  “the  contract  for  F-16  MTC  required 
the  services  to  be  integrated  into  the  F-16  Ready  Aircrew  Program.”  The 
Assistant  Secretary  stated  that  this  is  an  error  in  fact  because  the  Ready  Aircrew 
Program  accreditation  was  not  a  contractual  requirement.  The  Assistant  Secretary 
stated  that  the  statement  in  the  report  regarding  the  Ready  Aircrew  Program 
accreditation  appeared  to  be  a  misunderstanding  of  the  Air  Force’s  simulation 
certification  results  and  a  Ready  Aircrew  Program  letter  dated  December  5,  2005, 
and  requested  removal  of  all  references  to  Ready  Aircrew  Program  accreditation 
as  a  requirement.  The  Assistant  Secretary  provided  the  following  additional 
clarification: 

a.  The  government  conducts  a  simulator  certification  to  validate  training 
capability.  Simulated  events  are  evaluated  and  assigned  a  training  value  code.  A 
training  value  code  of  1,  2,  or  3  indicates  an  event  is  certified  for  formal  training 
(January  2005  simulator  certification  report,  table  1).  The  simulator  certification 
team  may  also  further  recommend  training  value  code  1  events  (fully  replicate  the 
aircraft)  for  accreditation  in  the  Ready  Aircrew  Program  as  prescribed  by  Air 
Force  Instruction  36-2248,  paragraph  5.5.1.  Air  Combat  Command/A3T  is  the 
authority  for  Ready  Aircrew  Program  certification,  and  while  they  carefully 
consider  simulator  certification  recommendations,  they  are  not  bound  by  them. 
The  contractor  has  no  control  on  Ready  Aircrew  Program  accreditation  of 
simulator  capabilities. 

b.  For  some  events,  a  training  value  code  1  rating  requires  simulation 
capabilities  that  are  not  available,  not  affordable,  or  unnecessary  for  the  role  or  a 
particular  MTC;  for  example,  the  Air  Force  intentionally  did  not  state 
requirements  for  capabilities  such  as  depth  perception,  20/20  resolution,  and 
simulated  motion.  This  meant  many  of  the  events,  and  therefore  mission,  would 
never  receive  a  training  value  code  1  rating.  While  the  contractor  must  ensure  all 
tasks  from  the  Training  Task  List  in  the  Statement  of  Work  placed  on  contract  are 
certified  for  fonnal  training  (training  value  code  1-3),  they  do  not  have  to  ensure 
all  tasks  receive  a  simulator  certification  recommendation  for  inclusion  in  the 
Ready  Aircrew  Program  (training  value  code  1).  Paragraph  3.0.6  of  the  Statement 
of  Work  requires  the  contractor  to  conduct  assessments  “to  the  level  necessary  to 
ensure  that  the  F-16  MTC  can  be  integrated  into  the  F-16  Ready  Aircrew  Program 
and  the  respective  training  syllabus,  as  applicable.”  The  above  is  clarification  to 
show  why  Ready  Aircrew  Program  accreditation  is  not,  and  could  not  be,  a 
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contractual  requirement.  It  is  therefore  not  appropriate  to  have  a  finding  that 
assesses  contractor  performance  based  on  Ready  Aircrew  Program. 

c.  In  assessing  contractor  performance,  the  simulator  certification 
demonstrated  89  percent  of  the  applicable  tasks  could  be  incorporated  into  the 
training  syllabus  (i.e.  certified  for  formal  training)  as  shown  in  the  January  2005 
simulator  certification  report,  table  2.2.  Although  not  a  contractual  requirement, 
the  following  explains  the  issue  of  Ready  Aircrew  Program  accreditation  for  the 
F-16  MTC.  Air  Combat  Command/A3T  conducted  an  additional  training  analysis 
and  detennined  credit  for  Ready  Aircrew  Program  requirements  could  be 
assigned  to  the  F-16  MTC.  However,  a  decision  was  made  in  April  2005  to  re¬ 
compete  the  contract  and  period  of  performance  for  current  services  was  set  to 
end  June  2007.  As  a  result,  they  did  not  mandate  Ready  Aircrew  Program  credit 
in  the  MTC,  with  a  concurrent  reduction  in  live  training  requirements,  because 
they  could  not  replace  the  live  training  if  there  was  a  break  in  service  between 
contracts. 

Audit  Response  to  Air  Force  Comments  on  Certification  of  F-16  MTC 
Simulation  Capabilities.  We  disagree  that  the  reference  to  the  Ready  Aircrew 
Program  is  an  error  in  fact  because  the  information  in  our  report  regarding  the 
Ready  Aircrew  Program  was  taken  directly  from  contractual  documents.  The 
Ready  Aircrew  Program  is  directly  referenced  in  the  contract  Statement  of  Work 
and  Integrated  Master  Plan.  Section  3.0.6  of  the  statement  of  work  specifically 
states  that  “The  Training  Capability  Requirements  Assessments  shall  be 
conducted  to  the  level  necessary  to  ensure  that  the  F-16  MTC  can  be  integrated 
into  the  Ready  Aircrew  Program  and  the  respective  training  syllabus,  as 
applicable.” 

This  statement  is  also  included  in  section  1 .2  of  the  Integrated  Master  Plan. 
Further,  attachment  4,  figure  1.3-1,  event  3,  of  the  Integrated  Master  Plan 
specifically  states  that  during  the  training  capabilities  requirements  assessment 
“the  MTC  team  will  also  agree  that  training  plans  and  objectives  correlate  with 
the  Air  Force  syllabus  and  the  Ready  Aircrew  Program.”  In  regards  to 
certification  of  MTC  services,  section  3.9.6  of  the  Statement  of  Work  states: 

The  Government  will  conduct  a  simulator  certification  on  all 
simulation  service  elements  to  validate  the  training  capability  of  the 
training  simulation  service.  Support  of  simulator  certification  shall 
include  personnel  to  operate  and  maintain  the  simulation  systems 
during  the  initial  and  annual  simulator  certification. 

Furthermore,  Air  Force  experts  create  simulator  certification  test  plans  and 
develop  mission  training  scenarios  based  on  Ready  Aircrew  Program 
requirements.  The  simulator  certification  is  used  to  evaluate  aircrew  training 
devices  to  determine  if  they  provide  credible  training  for  specific  flight  events. 

Payments  for  F-16  MTC  simulation  services  began  in  June  2002  at  Shaw  AFB. 
However,  the  initial  simulator  certification  report  was  not  issued  until  January 
2005.  The  simulator  certification  report  did  not  recommend  any  mission  events 
as  credible  to  Ready  Aircrew  Program  requirements.  Specifically,  the  simulator 
certification  overall  conclusion  states: 
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While  providing  satisfactory  support  for  normal  and  emergency 
procedures,  instalment  tasks,  element  coordination  and  air-to-ground 
procedural  training,  non-concurrency  with  MMC  3.3+  significantly 
restricts  training  effectiveness  for  mission  qualification  training  and 
continuation  training.  Additionally,  Two/Four-Ship  Flight  Lead 
Upgrade,  Instructor  Pilot  Upgrade,  Mission  Commander  Upgrade  and 
Continuation  Training  are  degraded  due  to  errors  in  the  visual 
environment  and  mission  rehearsal  training  cannot  be  accomplished 
due  to  the  lack  of  Distributed  Mission  Operations  capability. 

Air  Force  Comments  on  Effects  on  Mission  Training.  The  Assistant  Secretary 
of  the  Air  Force  (Acquisition)  disagreed  with  the  finding  on  the  effects  on  mission 
training  and  cited  Page  10  of  the  draft  report  which  states  that  the  MTC  did  not 
have  the  capability  to  train  14  of  the  16  missions  in  the  Statement  of  Work.  The 
Assistant  Secretary  stated  that  at  the  time  of  the  simulator  certification,  only  10  of 
the  16  missions  specified  in  the  Statement  of  Work  were  applicable  for  training  at 
Shaw  AFB  and  that  the  majority  of  tasks  in  these  10  missions  were  credited  for 
formal  training.  The  Assistant  Secretary  requested  that  the  report  accurately 
reflect  the  number  of  applicable  missions  and  include  an  assessment  of  trainable 
tasks.  Additional  details  were  provided  on  the  importance  of  assessing  trainable 
tasks  as  well  as  entire  missions. 

a.  Paragraph  1.2  in  the  Statement  of  Work  is  a  collective  list  of  the  16 
missions  specified  for  F-16  Block  40/50  MTCs.  Some  missions  are  not 
applicable  to  both  block  configurations  (Shaw  AFB  has  only  Block  50  aircraft). 
Additionally,  each  unit  trains  to  their  local  designated  operating  capability 
statement,  which  may  not  include  all  the  missions  for  a  given  configuration.  At 
Shaw  AFB  at  the  time  of  the  simulator  certification,  only  10  missions  were 
applicable.  The  6  non-applicable  missions  are  listed  below: 

1.  Low  Altitude  Navigation  &  Targeting  Infrared  for  Night  - 
applies  to  Block  40  configuration  only  (Performance  Specification,  paragraph 

3.2. 1.1) 

2.  Close  Air  Support  -  revising  aircraft  tactics  at  time  of  simulator 
certification,  therefore  did  not  evaluate. 

3.  Killer  Scout  -  not  a  contractual  requirement;  can  obtain 
capability  through  technology  insertion  (Statement  of  Work,  paragraph  1. 1.2.1) 

4.  Combat  Search  and  Rescue  -  not  a  contractual  requirement;  can 
obtain  capability  through  technology  insertion  (Statement  of  Work,  paragraph 

1. 1.2.1) 

5.  Night  Vision  Goggles  -  capability  added  to  contract  but  delivery 
date  was  several  months  after  the  simulator  certification. 

6.  Nuclear  Operations  -  Block  50  mission  but  not  performed  by 
Shaw  AFB  unit 

b.  Paragraph  3.2.2  of  the  Performance  Specification  states  that  the  MTC 
shall  “provide  the  capability  to  train  F-16  Training  Task  List  tasks  in 
accomplishment  of  the  F-16  missions  and  operations  listed  in  paragraph  3.2.1.” 
Assessing  the  number  of  trainable  tasks,  therefore,  is  a  more  accurate  reflection  of 
the  F-16  MTC’s  capabilities  and  the  contractor’s  technical  performance  than 
assessing  trainable  missions  on  an  all-or-nothing  basis. 
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c.  As  noted  earlier,  the  simulator  certification  conducted  at  Shaw  AFB  in 
October  2004  evaluated  74 1  applicable  training  tasks  and  certified  89  percent  for 
formal  training  (i.e.  they  received  a  training  value  code  rating  of  1,  2,  or  3). 
Deficiencies  of  the  MTC  at  the  time  prevented  it  from  simulation  6  percent  of  the 
tasks.  An  additional  4  percent  were  for  the  Night  Vision  Goggle  mission,  which 
wasn’t  yet  contractually  required,  as  noted  above.  [Note:  Since  October  2004,  the 
contractor  resolved  deficiencies  and  added  Night  Vision  Goggle  capability;  these 
tasks  are  ready  to  be  certified  at  the  next  simulator  certification.]  The  remaining 

1  percent  were  either  inconsistent  with  unit  doctrine  or  seldom  accomplished  by 
unit  pilots.  No  evaluated  tasks  were  rated  uncertified  for  formal  training  (training 
value  code  4).  These  results  found  in  paragraph  2.2.1  of  the  January  2005 
simulator  certification  report. 

d.  The  simulator  certification  also  included  pilot  evaluations.  Thirty-two 
pilots  evaluated  the  MTC’s  “capability... to  provide  advance  mission  training 
(weapons  employment,  mission  rehearsal,  tactics,  defensive  reactions,  etc.)”  Five 
of  them  rated  the  MTC  satisfactory,  23  rated  it  excellent,  and  four  rated  it 
outstanding  (January  2005  simulator  certification  report,  supplement  B). 

e.  From  very  early  in  its  fielding,  even  prior  to  the  simulator  certification, 
the  F-16  MTC  provided  a  great  deal  of  high-quality  training.  A  comment  from  an 
Operation  Iraqi  Freedom  F-16  pilot  on  21  March  2003  illustrates  this  well.  “I  was 
‘downtown’  at  the  start  of  tonight’s  activities. .  .The  cool  part  is  that  the  location 
of  my  flight  and  the  tactics  employed  were  exactly  like  we  were  practicing  in  the 
MTC  at  Shaw  AFB  before  we  left .  It  was  pretty  cool  to  see  the  terrain  and 
scenario  as  practiced.  Talk  about  mission  rehearsal.”  Today,  it  provides  even 
more  high-quality  training  to  our  pilots. 

Audit  Response  to  Air  Force  Comments  on  Effects  on  Mission  Training.  We 

acknowledge  that  the  Air  Force  has  received  some  additional  training  benefits 
with  the  F-16  MTC.  However,  the  Air  Force  is  not  receiving  the  services  for 
which  it  contracted.  The  Air  Force  states  that  a  majority  of  the  tasks  in  the  10 
missions  were  certified  for  formal  training.  However,  each  mission  requires 
subsets  of  training  tasks  performed  simultaneously.  During  the  audit,  we 
interviewed  the  Test  Director  for  the  29th  Training  Systems  Squadron  who  was 
responsible  for  the  only  official  simulator  certification  of  the  F-16  MTC.  The 
simulator  certification  Test  Director  stated  that  the  inability  of  a  small  number  of 
tasks  prevents  mission  training  capability.  Therefore,  while  pilots  may  obtain 
some  benefit  from  the  ability  to  train  specific  tasks,  the  pilots  are  unable  to  obtain 
full  mission  training. 

The  Air  Force  wrote  the  contract  for  F-16  MTC  in  such  vague  tenninology  that  it 
makes  it  extremely  difficult  to  objectively  evaluate  the  adequacy  of  simulation 
services  provided  by  the  contractor.  The  Air  Force  cites  training  value  codes  and 
percentages  of  applicable  tasks  that  were  certified  for  formal  training  but  none  of 
these  terms  are  discussed  or  defined  anywhere  in  the  contractual  documents.  In 
contrast,  the  statement  of  work  does  list  the  missions  that  were  to  be  trained  in  the 
F-16  MTC.  Specifically,  section  1.2  of  the  Statement  of  Work  states: 

The  Contractor  shall  confirm  the  MTC  training  task  list  with  Air 

Combat  Command.  The  Contractor  shall  define  the  core  typical 
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mission  training  profiles  and  scenarios  from  the  training  descriptions 
provided.  Training  activities  shall  include  Mission  Qualification 
Training,  Two/Four-Ship  Flight  Lead  Upgrade,  Instructor  Pilot 
Upgrade,  Mission  Commander  Upgrade,  and  Continuation  Training. 

Preliminary  objectives  shall  be  derived  for  future  Air  Expeditionary 
Force  Preparation  and  Mission  Rehearsal.  The  confirmed  missions  to 
be  trained  in  the  MTC  shall  include: 

a.  Defense  Counter  Air 

b.  Offensive  Counter  Air 

c.  Suppression  of  Enemy  Air  Defenses 

d.  Low  Altitude  Navigational  and  Targeting  Infrared  for  Night 

e.  Medium  Altitude  Operations 

f.  Tactics 

g.  Air  Interdiction 

h.  Weather  Approach  Procedures 

i.  Emergency  Procedures 

j.  Basic  Fighter  Maneuvers 

k.  Air  Combat  Maneuvers 

l.  Close  Air  Support 

m.  Killer  Scout 

n.  Combat  Search  and  Rescue 

o.  Night  Vision  Goggle  Operations 

p.  Nuclear  Operations 

The  Air  Force  states  that  at  the  time  of  the  simulator  certification  only  10  of  the 
16  mission  listed  in  the  statement  of  work  were  applicable  for  training  at  Shaw 
AFB.  Our  finding  addresses  the  fact  that  the  contract  listed  16  specific  missions 
to  be  trained  in  the  MTC  and  that  only  2  of  the  missions  were  certified  for  Ready 
Aircrew  Program  training  at  the  time  of  our  review.  If  there  are  missions  that 
were  not  applicable  during  the  simulator  certification  at  Shaw  AFB  we  would 
have  expected  to  see  them  addressed  during  a  simulator  certification  at  the 
applicable  MTC  site.  The  Air  Force  response  fails  to  acknowledge  that  the 
contract  required  a  simulator  certification  to  be  performed  as  each  MTC  site  was 
activated,  and  annually  thereafter.  Simulator  certifications  were  not  conducted  as 
sites  were  activated  and  none  of  the  four  operational  sites  have  successfully 
completed  an  annual  simulator  certification  as  required  by  the  contract.  At  the 
time  of  the  audit,  the  Air  Force  had  four  active  MTC  sites,  yet  only  one  simulator 
certification  has  been  completed.  This  was  the  initial  F-16  MTC  simulator 
certification  that  was  conducted  at  Shaw  AFB  in  October  2004  and  the  official 
report  was  not  issued  until  January  2005  (Note  that  payments  for  simulator 
services  began  in  June  2002).  The  missions  that  were  not  applicable  to 
Shaw  AFB  should  have  been  tested  during  a  simulator  certification  at  the 
applicable  site.  However,  these  simulator  certifications  were  never  accomplished. 

The  Air  Force  response  also  makes  reference  to  pilot  evaluations  included  in  the 
simulator  certification  report.  The  pilot  survey  fonn  is  a  subjective  evaluation, 
not  an  objective  overall  evaluation  of  the  simulator  and  its  capability  to  meet 
contract  requirements.  The  results  of  pilot  surveys  were  not  mentioned  anywhere 
in  the  contractual  documents  as  a  measure  of  contractor  performance. 
Additionally,  in  response  to  another  question  on  the  same  F-16  MTC  survey  eight 
of  thirty-one  pilots  rated  the  MTC  as  unsatisfactory  or  marginal  because  it  was 
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unable  to  start  a  scheduled  mission  on  time  and  complete  the  mission  without 
disruptions  such  as  computer  crashes  or  reboots.  The  January  2005  simulator 
certification  report  revealed  that  observed  mission  reliability  was  only  72  percent. 
In  addition,  the  simulator  certification  Test  Director  informed  us  that  poor  system 
stability  continues  to  be  a  problem  and  has  impacted  the  successful  completion  of 
simulator  certifications  at  other  operational  MTC  sites. 
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Appendix  E.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Director,  Acquisition  Resources  and  Analysis 
Under  Secretary  of  Defense  (Comp trailer)/ Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Director,  Program  Analysis  and  Evaluation 
Director,  Defense  Procurement  and  Acquisition  Policy 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 
Commander,  Air  Combat  Command 


Combatant  Command 

Inspector  General,  U.S.  Joint  Forces  Command 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Contract  Management  Agency 
Director,  Defense  Finance  and  Accounting  Service 
Director,  Defense  Intelligence  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 
Director,  Defense  Systems  Management  College 
Commander,  Defense  Contract  Management  Dayton  Ohio 
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Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 

Senate  Committee  on  Homeland  Security  and  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Anned  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency  and  Financial  Management,  Committee 
on  Government  Reform 

House  Subcommittee  on  National  Security,  Emerging  Threats,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Information  Policy,  Intergovernmental  Relations, 
and  the  Census,  Committee  on  Government  Refonn 
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Assistant  Secretary  of  the  Air  Force 
(Acquisition)  Comments 


DEPARTMENT  OF  THE  AIR  FORCE 
WASHINGTON  DC 


OFFICE  OF  TOE  ASSISI  ANT  SECRETARY 


2  5  SEP  2006 


MEMORANDUM  FOR  DEPUTY  INSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 


FROM:  SAF/AQ 

SUBJECT:  Report  on 
Simulation 


Acceptance  and  Surveillance  of  F-16  Mission  Training  Center  (MTC) 
l  Services;  August  2, 2006,  Report  Number  D2005FJ-0171.001 


1.  This  is  in  reply  to  your  memorandum  requesting  the  Assistant  Secretary  of  the  Ait  Force 
(Financial  Management  and  Comptroller)  provide  Air  Force  comments  on  the  subject  report, 

We  appreciate  the  opportunity  to  review  and  provide  comments. 

2.  We  concur  with  both  recommendations  and  have  begun  implementing  them  along  with  our 
own  lessons  learned  on  Training  Simulation  Service  contracts.  Detailed  responses  to  the 
recommendations,  along  with  our  implementation  plan,  are  included  in  Attachment  1. 

3.  However,  we  do  not  concur  with  some  of  the  findings,  primarily  on  F-1 6  MTC  certification 
and  effects  on  training.  We  believe  the  draft  report  leads  the  reader  to  incorrect  conclusions. 
While  there  is  room  for  improvement  in  verifying  contractor  compliance,  the  Air  Foi(ce  is 
receiving  a  far  greater  amount  of  capability  than  the  report  indicates.  Attachment  2  (hows  this 
by  providing  additional  information  on  contractual  requirements.  I  request  your  consideration  in 
revising  this  information  prior  to  publishing  the  final  report. 

4.  My  staff  stands  ready  to  provide  additional  information  and  to  work  with  you  on  your  final 
report  preparations.  My  point  of  contact  is  Major  Tim  Dickinson  at  703-588-7724. 


SUE  C.  PAYTON 
Assistant  Secretary  of  the  Air  Force 
(Acquisition) 


2  Attachments: 

1.  Air  Force  Response  to  DOD  1G  Recommendations 

2.  Air  Force  Non-concurrence  with  DOD  IG  Findings 


Attachment  1:  Air  Force  Response  to  DOD  IG  Recommendations 

DOD  IG  Draft  Report,  “Acceptance  and  Surveillance  of  F-16  Mission  Training  Center 
Simulation  Services;”  August  2,  2006,  Report  Number  D2005FJ-0171.001 


RECOMMENDATION  1:  The  DOD  IG  recommends  that  the  Director,  Simulator  Systems 
Group,  Aeronautical  Systems  Center,  coordinate  with  Air  Combat  Command  and  Defense 
Contract  Management  Agency  (DCMA)  personnel  to: 

a.  Analyze  the  existing  capabilities  of  each  F-16  Mission  Training  Center  (MTC)  site  against 
contracted  requirements  and  reduce  service  payments  through  the  end  of  the  contract  in  June 
2007  for  any  requirements  not  provided. 

AF  RESPONSE:  The  Air  Force  will  reduce  service  payments  to  the  contractor  if  the  F-16  MTC 
simulation  services  they  provide  do  not  meet  contractual  requirements.  This  approach  was  taken 
in  the  past  and  will  remain  part  of  our  contract  management  strategy.  The  677  Aeronautical 
Systems  Group  (677  AESG,  formerly  the  Simulator  Systems  Group),  in  association  with  DCMA, 
will  monitor  the  contractor’s  performance  to  ensure  the  Air  Force  is  receiving  all  contractually 
required  services. 

b.  Require  contractors  to  demonstrate  that  simulation  services  meet  all  statement  of  work 
requirements  prior  to  accepting  the  service. 

AF  RESPONSE:  Additional  services  are  not  planned  for  the  current  F-16  MTC  simulation 
service  contract.  For  future  simulation  service  contracts,  the  677  AESG  will  ensure  all 
contractual  requirements  are  met  prior  to  acceptance  of  the  service. 

c.  Formalize  the  test  discrepancy  reporting  and  resolution  process  for  simulation  service 
contracts.  The  process  should  include  a  description  of  each  discrepancy  (problem  that  prevents 
the  simulator  from  performing  specific  functions  essential  for  mission  training)  and  the  date  each 
discrepancy  is  identified,  tested,  and  resolved. 

AF  RESPONSE:  The  request  for  proposal  for  the  follow-on  F-16  MTC  simulation  service 
contract  identifies  detailed  requirements,  including  those  recommended  above,  for  a  formalized 
deficiency  reporting  system. 

d.  Develop  quality  assurance  surveillance  plans  in  conjunction  with  the  statement  of  work  for 
all  simulation  service  contracts  as  required  by  Federal  Acquisition  Regulations. 

AF  RESPONSE:  DCMA  is  currently  drafting  a  quality  assurance  surveillance  plan  for  the  F-16 
MTC  site  at  Mt.  Home  AFB.  Quality  assurance  surveillance  plans  are  complete  for  all  other  F- 
16  MTC  sites.  The  677  AESG  is  working  closely  with  DCMA  to  develop,  in  accordance  with 
the  FARs,  quality  assurance  surveillance  plans  for  the  follow-on  F-16  MTC  simulation  service 
contract. 
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RECOMMENDATION  2:  The  DOD  IG  recommends  that  the  Commander,  Operations  and 
Training  Branch,  Headquarters  United  States  Air  Force  update  Air  Force  instructions  to: 

a.  Identify  roles  and  responsibilities  for  the  management  and  administration  of  simulation 
service  contracts. 

AF  RESPONSE:  The  Director,  Current  Operations  and  Training,  Headquarters  United  States 
Air  Force  (AF/A30),  is  revising  AFI  36-225 1  “Management  of  Air  Force  Training  Systems”  to 
more  clearly  define  the  roles  and  responsibilities  of  personnel  who  manage  and  administer 
simulation  service  contracts  throughout  all  major  commands.  Headquarters  Air  Combat 
Command  (ACC/A30)  is  doing  the  same  for  the  combat  air  forces  in  AFI  36-2248  “Operation 
and  Management  of  Aircrew  Training  Devices.”  The  planned  completion  for  these  revisions  is 
31  December  2006. 

b.  Mandate  training  for  quality  assurance  personnel  performing  oversight  of  simulation 
service  contracts. 

AF  RESPONSE:  The  revisions  to  AFI  36-225 1  and  AFI  36-2248  will  also  mandate  training  for 
Air  Force  personnel  when  a  DCMA  contracting  officer  delegates  quality  assurance  duties  to 
them. 
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Introduction.  The  capabilities  of  all  our  Mission  Training  Centers  (MTC)  continue  to  improve, 
as  does  our  management  of  training  simulation  service  contracts.  In  the  case  of  the  current  F-16 
MTC  contract,  the  Air  Force  agrees  a  better  job  of  testing  and  surveillance  should  have  been 
done  to  ensure  requirements  were  met  before  simulation  services  began.  All  elements  of  the 
service  should  have  been  tested  and  the  essential  discrepancies  resolved  prior  to  equipment 
leaving  the  contractor's  facility.  However,  the  Air  Force  disagrees  with  the  finding  that  stated, 
"The  Mission  Training  Centers  did  not  simulate  14  of  the  16  missions  specified  in  the  contract.  " 
as  noted  in  the  Executive  Summary.  While  it  took  us  longer  than  planned,  the  Air  Force  is 
receiving  a  far  greater  amount  of  capability  than  the  report  indicates.  There  are  several  elements 
to  this  issue  that  are  not  considered  in  the  draft  report  that  we  believe  should  be  included  in  the 
final  report. 

FINDING:  Certification  of  F-16  MTC  Simulation  Capabilities.  Page  8  of  the  report  slates, 
“The  contract  for  F-16  MTC  .  .  . required  the  services  to  be  integrated  into  the  F-16  ready  aircrew 
program.” 

AF  RESPONSE:  This  is  an  error  in  fact  because  RAP  accreditation  is  not  a  contractual 
requirement.  The  statement  appears  to  be  a  misunderstanding  of  the  Air  Force’s  simulation 
certification  (S1MCERT)  results  and  a  RAP  letter  dated  5  December  2005.  We  request  removal 
of  all  references  to  RAP  accreditation  as  a  requirement.  Additional  clarification  is  provided 
below. 

a.  The  government  conducts  a  SIMCERT  to  validate  training  capability.  Simulated 
events  are  evaluated  and  assigned  a  Training  Value  Code  (TVC).  A  TVC  of  1,  2,  or  3  indicates 
an  event  is  certified  for  formal  training  (SIMCERT  Report,  Table  1).  The  SIMCERT  team  may 
also  further  recommend  TVC  1  events— fully  replicate  the  aircraft—for  accreditation  in  the  RAP 
(AFI  36-2248,  paragraph  5.5.1).  ACC/A3T  is  the  authority  for  RAP  certification,  and  while  they 
carefully  consider  SIMCERT  recommendations,  they  are  not  bound  by  them.  The  contractor  has 
no  control  on  RAP  accreditation  of  simulator  capabilities. 

b.  For  some  events,  a  TVC  I  rating  requires  simulation  capabilities  that  are  not  available, 
not  affordable,  or  unnecessary  for  the  role  of  a  particular  MTC;  for  example,  the  Air  Force 
intentionally  did  not  state  requirements  for  capabilities  such  as  depth  perception,  20/20 
resolution,  and  simulated  motion.  This  meant  many  of  the  events,  and  therefore  missions,  would 
never  receive  a  TVC  1  rating.  While  the  contractor  must  ensure  ail  tasks  from  the  Training  Task 
List  in  the  Statement  of  Work  (SOW)  placed  on  contract  are  certified  for  formal  training  (TVC 
1-3),  they  do  not  have  to  ensure  all  tasks  receive  a  SIMCERT  recommendation  for  inclusion  in 
the  RAP  (TVC  1).  Paragraph  3.0.6  of  the  SOW  requires  the  contractor  to  conduct  assessments 
“to  the  level  necessary  to  ensure  that  the  F-16  MTC  can  be  integrated  into  the  F-16  RAP  and  the 
respective  training  syllabus,  as  applicable .”  The  above  is  clarification  to  show  why  RAP 
accreditation  is  not,  and  could  not  be,  a  contractual  requirement.  It  is  therefore  not  appropriate  to 
have  a  finding  that  assesses  contractor  performance  based  on  RAP. 

c.  In  assessing  contractor  performance,  the  SIMCERT  demonstrated  89%  of  the 
applicable  tasks  could  be  incorporated  into  the  training  syllabus  i.e.  certified  for  formal  training 
(SIMCERT  Report,  Table  2.2).  Although  not  a  contractual  requirement,  the  following  explains 
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the  issue  of  RAP  accreditation  for  the  F-16  MTC.  ACC/A3T  conducted  an  additional  training 
analysis  and  determined  credit  for  RAP  requirements  could  be  assigned  to  the  F-16  MTC. 
However,  a  decision  was  made  in  Apr  05  to  re-compele  the  contract  and  period  of  performance 
for  current  services  was  set  to  end  Jun  07.  Asa  result,  they  did  not  mandate  RAP  credit  in  the 
MTC,  with  a  concurrent  reduction  in  live  training  requirements,  because  they  could  not  replace 
the  live  training  if  there  was  a  break  in  service  between  contracts. 

FINDING:  Effects  on  mission  training.  Page  10  of  the  report  indicates  the  MTC  did  not  have 
the  capability  to  train  1 4  of  the  1 6  missions  in  the  SOW. 

AF  RESPONSE:  At  the  time  of  the  S1MCERT,  only  10  of  the  16  missions  listed  in  the  SOW 
w  ere  applicable  for  training  at  Shaw  AFB.  The  majority  of  tasks  in  these  10  missions  were 
certified  for  formal  training.  We  request  the  report  accurately  reflect  the  number  of  applicable 
missions  and  include  an  assessment  of  trainable  tasks.  The  importance  of  assessing  trainable 
tasks  as  well  as  entire  missions  is  described  below. 

a.  Paragraph  1.2  in  the  SOW  is  a  collective  list  of  the  16  missions  specified  for  F-16 
Block  40/50  MTCs.  Some  missions  are  not  applicable  to  both  block  configurations  (Shaw  AFB 
has  only  Block  50  aircraft).  Additionally,  each  unit  trains  to  their  local  designated  operating 
capability  statement,  which  may  not  include  all  the  missions  for  a  given  configuration.  At  Shaw 
AFB  at  the  time  of  the  SIMCERT,  only  10  missions  were  applicable.  The  6  non-applicable 
missions  are  listed  below; 

1 .  Low  Altitude  Navigation  &  Targeting  Infrared  for  Night  (LANTIRN)  -  applies  to 
Block  40  configuration  only  (Performance  Specification,  paragraph  3.2. 1.1) 

2.  Close  Air  Support  -  revising  aircraft  tactics  at  time  of  SIMCERT,  therefore  did 
not  evaluate 

3.  Killer  Scout  -  not  a  contractual  requirement;  can  obtain  capability  through 
lechnolmrv  insertion  iSOW.  naravranh  1.1.7.  11 
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capability;  these  tasks  are  ready  to  be  certified  at  the  next  SIMCERT.]  The  remaining  1%  were 
either  inconsistent  with  unit  doctrine  or  seldom  accomplished  by  unit  pilots.  No  evaluated  tasks 
were  rated  uncertified  for  formal  training  (TVC  4).  These  results  are  found  in  paragraph  2.2. 1  of 
the  S1MCERT  report. 

d.  The  SIMCERT  also  included  pilot  evaluations.  Thirty-two  pilots  evaluated  the  MTC’s 
“capability... to  provide  ADVANCED  MISSION  training  (weapons  employment,  mission 
rehearsal,  tactics,  defensive  reactions,  EW,  etc.)”  Five  of  them  rated  the  MTC  satisfactory,  23 
rated  it  excellent,  and  four  rated  it  outstanding  (SIMCERT  Report,  Supplement  B). 

e.  From  very  early  in  its  fielding,  even  prior  to  the  SIMCERT,  the  F-16  MTC  provided  a 
great  deal  of  high-quality  training.  A  comment  from  an  Operation  Iraqi  Freedom  F-16  pilot  on 
21  March  2003  illustrates  this  well.  “/  was  ‘downtown  ’  at  the  start  of  tonight's  activities...  The 
cool  part  is  that  the  location  of my  flight  and  the  tactics  employed  were  EXACTL  Y  like  we  were 
practicing  in  the  MTC  at  Shaw  before  we  left.  It  was  pretty  cool  to  see  the  terrain  and  scenario 
as  practiced.  Talk  about  mission  rehearsal Today,  it  provides  even  more  high-quality  training 
to  our  F-16  pilots, 

Summary.  The  draft  report  findings  mentioned  above  do  not  provide  an  accurate  assessment  of 
F- 1 6  MTC  capabilities.  One  finding  in  particular-rAe  MTC  did  not  simulate  14  of  the  16 
missions— leads  the  reader  to  conclude  the  Air  Force  achieved  very  little  return  on  investment. 
While  there  is  room  for  improvement  in  verifying  contractor  compliance,  which  we  are  working 
on,  the  Air  Force  achieved  a  far  greater  amount  of  capability  than  the  report  indicates.  The  F-16 
MTC  has  been  providing  significant  training  benefits  over  the  older  devices  since  early  2003  and 
today  numerous  tasks  are  prepared  for  incorporation  into  the  F-16  Ready  Aircrew  Program.  We 
appreciate  your  consideration  in  updating  this  information  ill  your  final  report. 
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MEMORANDUM  FOR  OFFICE  OF  THE  DEPUTY  INSPECTOR  GENERAL  FOR 
AUDITING  OF  THE  DEPARTMENT  OF  DEFENSE, 
DEFENSE  FINANCIAL  AUDITING  SERVICE 


SUBJECT:  DCMA  Management  Comments,  Report  on  the  Acceptance  and 
Surveillance  of  F-16  Mission  Training  Center  Simulation  Services 
(Project  No  D2005-DOOOFJ-0171.001) 


Defense  Contract  Management  Agency  (DCMA)  Dayton  takes  no  exceptions  to  any 
of  the  findings  contained  in  the  draft  report,  and  the  report  contains  no 
recommendations  directly  addressed  to  our  organization  However,  in  our  role  as 
the  single  DCMA  office  responsible  for  contract  administration  services  for  the  Air 
Force  Simulator  Systems  Group  (SMSG),  we  can  take  this  occasion  to  review  any 
possible  opportunities  for  improvement  in  our  support  of  SMSG  in  particular,  and  in 
the  administration  of  Simulator  contracts  in  general 

From  our  review  and  analysis  of  the  draft  report,  we  have  identified  for  improvement 
certain  DCMA  Dayton  internal  processes  in  the  areas  of  site  reporting  procedures, 
lines  of  communication,  and  training.  We  have  taken  action  on  these  to  improve 
our  level  of  support  to  SMSG  not  only  for  the  F-16  program,  but  all  SMSG 
programs  We  are  currently  rewriting  our  Memorandum  of  Agreement  with  SMSG 
to  incorporate  these  process  changes,  and  formalize  our  support  in  assisting 
SMSG,  and  the  Air  Force,  in  meeting  their  requirements 

We  appreciate  the  opportunity  to  comment  on  this  draft  report  If  you  have  any 
questions  or  require  additional  information,  please  contact  me  at  937-656-3079, 
email  james  cox@dcma  mil.  / 


JAMES  D  COX 
I  Deputy,  Operations  Group 
DCMA  Dayton 
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